Re: [-empyre-] Missive 1: Reply to Annabeth.



Considering audio in a broader MUVE perspective.

I am looking forward to in-engine DSP (Digital Signal Processing, which Annabeth referred to briefly, is the process of generating and/ or effecting sounds), although it seems a long way off.

While this will enable such techniques as automatic reverb based on the programmed physics of the space (eg, a large rock cavern or a beach), it will also enable the true post-convergent approach of "playing the space" as an instrument. Gestures, position, scale, velocity, colour, network data, etc, will all act as manipulable parameters in the generation and effecting of sound, which in turn will effect these parameters in a symbiotic feedback loop.

Currently, John McCormick and I are investigating this idea by streaming data from realtime 3D environments into Pure Data (pd, an open source more powerful version of Max/MSP) for DSP, then back into the realtime 3D environment. It works, but it's clumsy, and trying to do it in SL (without hacking the client) is clumsier, although we are working on that with our realworld controller project. John and I will be performing some of this stuff in Melbourne in December this year and attempting to have it accessible it via the network.

Like many of the concepts we discuss in this forum, it is conceptually easy to grasp, but the technology seriously lags. Some work is being done on procedural generation of sound in game engines, but it is still in its infancy. Now that the fetishistic obsessions of the graphics card industry is settling a little, more attention will be paid to audio.

In non-game engine terms, Niall Moody is doing interesting work in this area. He's developed the Helian X3D browser, that is specifically aimed at realtime 3D audio:

http://www.niallmoody.com/heilan/index.htm

Also, Fabio and I, along with a small group of researchers at RMIT, are working with Multiverse, which will likely offer more potential in this field.

Adam


On 10/08/2007, at 8:47 AM, Adam Nash wrote:

The Port is a great place, and Annabeth's work in particular is consistently inventive and inspiring, investigating and manipulating the environment on its own terms, rather than for its representative qualities.

Continuing with the discussion of sound within SL, Annabeth is so right when she says that sound is overlooked in SL. This is historically true of all realtime 3D environments, but SL is probably the worst of them all. We're currently stuck with 10 sec digital sample files and unpredictable (apparently undefined) behaviour for both pre-loading and falloff. Further, we have no access to the end-user's prefs for doppler, falloff etc. And of course there is the technical philosophy of the SL engine that vision will take precedence over sound whenever a choice has to be made.

On the bright side, this forces a different approach, one that involves going with these restrictions/unpredictabilities, and this is what I try to do in my SL audiovisual pieces. It is possible to create small scale audiovisual works that respond to avatar input, or other inworld data input such as weather, object proximity etc, or that display some programatic behaviour.

On a side note, Annabeth, have you noticed an empirical limit to the number of sounds that can be simultaneously playing? I've found it to be around 40 - most frustrating!

Adam

On 10/08/2007, at 3:42 AM, Anna Robinson wrote:

Hi Patrick and All,

Taking advantage while Second Life grid is having 'connectivity' issues.

" Annabeth, what has been your experience in using spatialized sound in
the SL environment, and what issues have you come upon in regards to
visitor experience, and do you feel that sound works differently in SL
as a medium at all? "


Uploading sound files into Second Life have the problem of being
limited to 10sec lengths.
So when creating sound scapes, you have to be creative within this
limitation. You can have sound loops, event sounds ( when I mean event
- sound fx triggered on touch, time, collision etc ), and scripted
sequencers - ( objects that play a sound files one after another , or
randomly )
Mixing this all together you can create quite rich soundscapes, that
also can be unique each time you visit the second life area.
Apart from adjusting volume, you can't signal process the sound once
its in SL. So if you want a reverberant sound for inside a cave, you
have to do that to the sound before uploading. If you are making a
musical instrument, you need to make a sound file for each note, as
you can change the pitch once inworld.



It's a bit frustrating that I'm in the middle of a large corporate project - ( one of the reasons I've been quiet ), and I'm under a non-disclosure act - so can't talk deeply about it.

The project has quite a large sound plan, ranging from atmospheric
sounds that change with the course of a Second Life day, to spot fx
that are triggered via Avatar interaction or narrative thread that
goes through the experience.

One of the problems we've encountered with working sound across an
entire sim, is sound spillage. Too many emitters it can become a
cacophony, or sound specific to one area leaks into another space.
There are techniques where you can hard cut sound spatialy so it
limited within an area, but ultimately whats been attempted, is that
as you walk around the islands - the sound fades in and out gracefully
as you approach and leave locations, as it would in Real life.
So a kind of 3D sound mixing has to be done to achieve this effect,
and within second life you can change the volume of a specific
emitting sound.


Another issue that effects this scaping, is the soundfile needs to be
part of an object (prim), if it is to be played. The Second Life
Client software has a tendency to remove small objects from render if
you walk or turn away from them. As sound - 'for a better word' - is
sticky to the object, if the object disappears, so the sound cuts off
to. So a way to get around this is to create huge invisible objects
that center on where the sound should originate.


Another big problem is waiting for sounds to cache for the vistor,
there are a few tricks to speed up this process, but it has a great
knock on effect to 'one-off' interaction sound fx,  ie. you touch a
button, but it can take a second or two to hear the click. So you
don't get a positive audio response to your interaction. Sometimes
this lag can take longer. I've had situations with my own sound
sculptures where avatars have played for a several minutes, without
sounds caching.
This can frustrate people, so you have to either force preload sounds
or have some info - saying 'please be patient'.

When it does all work together - having this bed of atmospheric sound
really glues the visual and emotional experience together, as much as
soundscaping has an effect on film and television. Its a sorely
overlooked part of the Second Life experience.

....


I think the addition of spatialised Voice Chat to SL recently, has created some interesting opportunities for performative works in Second Life, and I think that's where Virtual World Audio could be unique.

With Streaming audio into second life, it has no origin point within
the environment, it plays equal volume across the area its designated
to play within - sounding somewhat like having Itunes running
concurrently while you are inworld. It is also a mixed-down channel.

With the voice chat - each avatar can become a seperate channel of
streaming audio that can be spatially placed within second life. This
merging of a telepresent group performance and architectural space,
could be really exciting. It doesn't have to be voice, the avatar can
be a conduit for a sound feed from instruments or other sound sources.


---

And yep the Port has been overlooked a lot, though that's because it
has been quiet for several months... so if any of you do venture into
Second Life, make sure you visit.


Anna
_______________________________________________
empyre forum
empyre@lists.cofa.unsw.edu.au
http://www.subtle.net/empyre
_______________________________________________
empyre forum
empyre@lists.cofa.unsw.edu.au
http://www.subtle.net/empyre



This archive was generated by a fusion of Pipermail 0.09 (Mailman edition) and MHonArc 2.6.8.